class: center, middle # The Unix/Linux Resource Collection <br> ## Working in the Linux Command-Line: ### A Short List of the Best and Greatest Commands .author[ Stewart Weiss <br> ] .license[ Copyright 2020-24 Stewart Weiss. Unless noted otherwise all content is released under a [Creative Commons Attribution-ShareAlike 4.0 International License](https://creativecommons.org/licenses/by/4.0/). ] <!--TOC_START --> --- name: toc ### Table of Contents [Overview](#overview)<br> [The Shell versus the Commands](#shell_vs_commands)<br> [A Bit about Shells](#about_shells)<br> [Command Navigation](#command_navigation)<br> [Command Editing](#command_editing)<br> [Introduction to Commands](#command_intro)<br> [Files and Directories](#file_system)<br> [Special Directories: The Home Directory](#home_dir)<br> [The Current Working Directory](#working_dir)<br> [Working in Multiple Terminals](#many_shells)<br> [Listing Directory Contents](#listing_dirs)<br> [About File names](#filenames)<br> [Relative Pathnames](#pathnames)<br> [Navigating Directories](#navigating_dirs)<br> [Types of Commands](#command_types)<br> [Learning About Commands](#learning_commands)<br> [Getting Help: The Manual Pages](#man_pages)<br> [Searching Through Man Pages](#man_page_searching)<br> [The Info System](#info)<br> [Help for Shell Builtins](#bash_help)<br> [Categories of Commands](#command_categories)<br> [More Commands](#misc_commands)<br> --- ### Table of Contents [Command Syntax](#cmmd_syntax)<br> [Simple Command Syntax](#cmmd_syntax2)<br> [Complex Command Syntax](#cmmd_syntax3)<br> [Miscellaneous Commands](#misc_commands)<br> [Miscellaneous Commands](#misc_commands2)<br> [The Power of I/O Redirection](#redirection_examples1)<br> [More Examples of Redirection](#redirection_examples2)<br> [More Examples of I/O Redirection](#redirection_examples3)<br> [Streams, Files, and Redirection](#streams)<br> [File Descriptors and Streams](#file_descriptors)<br> [I/O Redirection](#io_redirection)<br> [Input Redirection](#input_redirection)<br> [Output Redirection](#output_redirection)<br> [Error Redirection](#error_redirection)<br> [Redirection Examples](#redirection_examples)<br> [Appending](#appending)<br> [Pipes](#pipes)<br> [Viewing Files](#viewing_files)<br> [`ls`](#ls)<br> [`basename`](#basename)<br> [`dirname`](#dirname)<br> [`find`](#find)<br> [finding by file name](#find_by_name)<br> [finding by time stamp](#find_by_time)<br> --- ### Table of Contents [finding by other properties](#find_misc)<br> [finding by other properties](#find_misc2)<br> [taking actions when files are found](#find_actions)<br> [Filter Introduction](#filter_intro)<br> [Filter Synopsis](#filter_synopsis)<br> [`cat` and `tac`](#cat_tac)<br> [`head` and `tail`](#head_tail)<br> [`sort`](#sorting)<br> [`sort` Examples (1)](#sort_examples1)<br> [`sort` Examples (2)](#sort_examples2)<br> [`uniq`](#uniq)<br> [Combining Filters](#combining_filters)<br> [Filters: `grep`, `egrep`, and `fgrep`](#grep_family)<br> [Regular Expression Rules](#regex1)<br> [Regular Expression Rules (2)](#regex2)<br> [Regular Expression Character Classes](#regex3)<br> [Regular Expression Character Classes (2)](#regex4)<br> [Regular Expressions: Anchors](#anchors)<br> [Regular Expressions: Backreferences](#backreferences)<br> [Filters: `cut` and `paste`](#cut_paste1)<br> [Filters: `cut` and `paste` (2)](#cut_paste2)<br> [Filters: `cut` and `paste` (3)](#cut_paste3)<br> [`awk`](#awk1)<br> [`awk` (2)](#awk2)<br> --- ### Table of Contents [`awk` (3)](#awk3)<br> [`awk` (4)](#awk4)<br> [`awk` (5)](#awk5)<br> [`awk` (6)](#awk6)<br> [`awk` (7)](#awk7)<br> [Filters yet to come:](#more_filters)<br> [Useful Links](#links)<br> [Exercises 1](#exercise1)<br> [Exercises 2](#exercise2)<br> [Exercises 3](#exercise3)<br> [Exercises 4](#exercise4)<br> [Exercises 5](#exercise5)<br> [Exercises 6](#exercise6)<br> [Exercises 7](#exercise7)<br> <!--TOC_END --> --- name: overview ## Overview - There are many different Unix operating systems. .redbold[Linux].super[1] is one of them. Other popular forms of Unix include BSD variants such as .redbold[FreeBSD], proprietary versions such as IBM's .redbold[AIX] and Oracle's .redbold[Solaris], and .redbold[Android]. - Most of what is in these slides is true of all "flavors" of UNIX, but some of it is only true of what is commonly called .redbold[GNU/Linux], which is an operating system consisting of a Linux kernel with a very rich and powerful set of tools that integrate into it, developed by the .redbold[GNU Project] with the support of the .redbold[Free Software Foundation] .super[2]. .footnote[ 1. Technically, Linux is the name of the .bluebold[kernel], the core of the operating system that is always memory-resident. 2. The GNU Project calls GNU/Linux the .redbold[GNU Operating System]. ] - There is too much to cover about working in the Unix command-line in a short slide presentation. - These slides cover many topics, none in great depth. But they are designed to cover the most important aspects of working in the .greenbold[UNIX programming environment]. --- name: shell_vs_commands ## The Shell versus the Commands - Working in the command-line really means doing two different things: - using the .greenbold[shell] - entering Unix .greenbold[commands] - You cannot enter any commands without using the shell, so you have to know something about using a shell program, such as `bash`. - But knowing `bash` and nothing more is pretty useless, because the purpose of using `bash` is to make it easy to do your work, which almost always involves executing commands. - These slides begin with light coverage of `bash`, the most common Linux shell.super[1], but primarily focus on Linux commands. A different set of slides covers `bash` scripting. - You are encouraged to read the many resources on the topics presented here. The last slide has links to the ones which, I think, are good and concise. .footnote[ 1. Other common open source shells include the .redbold[C shell] (`csh`), the .redbold[Bourne shell] (`sh`), and the .redbold[T C shell] (`tcsh`). ] --- name: about_shells ## A Bit about Shells - You need to know a bit about shells before you can enter any commands in Unix, so we begin with the very basics. - A .bluebold[shell] is an interactive command-line interpreter for the operating system, but it is also a programming language. - A .bluebold[command-line interpreter] is a program that displays a prompt (such as '$') and waits for you to type a command. When you type the command followed by a .greenbold[newline] character, it "executes" that command and then redisplays the prompt and starts this cycle all over again. - Example Using the `whoami` command: ```bash $ whoami stewart ``` `whoami` is an unusual command - it displays your username (just in case you forgot.) - The prompt can be customized in many ways.super[1]. On the system in which I have prepared these slides, whose name is `harpo`, my prompt is ```bash [stewart@harpo slides]$ ``` .footnote[ 1. The PS1 shell variable in `bash` controls the .greenbold[first level prompt string]. ] --- name: command_navigation ## Command Navigation - Modern shells, including `bash`, have a .greenbold[history mechanism] that saves the sequence of your most recent commands in a file and that allows you to recall and execute previous commands.super[1]. - The up and down arrow keys on the keyboard let you navigate this history: - up-arrow displays the command before the current one, - down-arrow displays the command after the current one. -- - .redbold[Example]. I enter the following three commands, the last two of which are just gibberish: ```bash [stewart@harpo ~]$ printenv HISTSIZE 500 [stewart@harpo ~]$ nonsense nonsense: command not found [stewart@harpo ~]$ whatsup whatsup: command not found [stewart@harpo ~]$ ``` -- If, in the current state, I press the up-arrow key 3 times, I can re-execute the command `printenv HISTSIZE`. .footnote[ 1. History is a bit complicated, but roughly, the default is 500 previous commands are saved in persistent storage. ] --- name: command_editing ## Command Editing - Modern shells, including `bash`, have a .greenbold[command editing] that allows you to use the left-arrow and right-arrow keys to move the cursor in the current command and then modify it by inserting or removing characters. - .redbold[Example]. I first type the following command but DO NOT press the .greenbold[ ENTER ] key (henceforth called .greenbold[newline]): ```bash [stewart@harpo ~]$ printenv HISTSIZE ``` -- I now move my cursor so that it is where the vertical bar is here: ```bash [stewart@harpo ~]$ printenv HIST|SIZE ``` and type the word `FILE`, delete the letters `SIZE` and do not yet enter newline: ```bash [stewart@harpo ~]$ printenv HISTFILE ``` -- When I enter the newline this is what I see: ```bash [stewart@harpo ~]$ printenv HISTFILE /home/stewart/.bash_history ``` --- name: command_intro ## Introduction to Commands - Unix commands are the tools that you use to get things done in Unix. You type their .greenbold[names] at the shell prompt, and they do their magic. - .redbold[Examples]. ```bash $ date Thu Mar 28 11:22:45 EDT 2019 $ echo "Hello world. I try to code responsibly!" Hello world. I try to code responsibly! $ wc ./README 72 464 2677 ./README ``` - The `date` command displays the current time and date; - `echo` displays the strings that follow it on the command-line; - `wc` is short for __w__ord __c__ount; it displays the number of lines, words, and characters in the file(s) whose names follow it on the command-line. --- name: file_system ## Files and Directories - Files reside in a large tree called the .greenbold[directory hierarchy]. Each has an .greenbold[absolute pathname], which is a string that defines the path from the root of the tree (which is called `/`) to the file. - A .bluebold[directory] is a special type of file that contains a table with a row for each file in it, and a reference that can be used to find the actual file on the system's storage devices.super[1]. .left-column2-bigright[ <img src="img/filesystem.png" height=360 style="float:center;" > ] .right-column2-bigright[ The absolute pathname of the `README` file in the representation of part of a file system to the left is ```bash /data/biocs/b/student.accounts/cs395.86/scratch/README ``` The absolute pathname to the directory `cs395.86` in this same file system is ```bash /data/biocs/b/student.accounts/cs395.86 ``` ] .footnote[ 1. Or if it is not a "real" file, then wherever the file is located. ] --- name: home_dir ## Special Directories: The Home Directory - The shell maintains two special directories for you: the .greenbold[home directory], and the .greenbold[current working directory]. - The .bluebold[home directory] is the one whose contents you see when you first open a terminal window or when you first login to the Unix system. - It is an abstraction that represents the place where all your personal stuff is supposed to be kept, like your home. Not in the street or some other public place! - Your home directory can be referred to by the special notation `~`. The home directory of a user whose name is `severus` would have the shorthand name `~severus`. --- name: working_dir ## The Current Working Directory - When you are working in a shell, at any moment in time you have a unique, special directory called your .greenbold[current working directory] or your .greenbold[present working directory]. - In these slides, we call it simply the .greenbold[current directory]. -- - The current directory is where you "are" in the file system at the moment, i.e., the directory that you feel like you are currently "in". -- - The `pwd` command displays the absolute pathname of your current directory: ```bash [stewart@harpo ~]$ pwd /home/stewart ``` - A single period `.` is a shorthand notation that represents the current working directory. -- - The shell keeps track of your current directory as you move about the file system in a special variable it maintains, named `PWD`. You can use `printenv` to see its current value: ```bash [stewart@harpo ~]$ printenv PWD /home/stewart ``` --- name: many_shells ## Working in Multiple Terminals - If you have two terminal windows open, you are running two copies of the shell and each copy will have its own squirreled away copy of the `PWD` variable.super[1]. - The two commands below were run in two different terminal windows open at the same time: .left-column2[ ```bash [stewart@harpo cs395.86]$ printenv PWD /home/stewart/hunter/cs395.86 ``` ] .right-column2[ ```bash [stewart@harpo cs340]$ printenv PWD /home/stewart/hunter/cs340 ``` ] .below-column2[ - The `pwd` command would show the same output. ] .footnote[ 1. This should suggest to you that each shell stores its variables in memory, not in a file. But Linux has a virtual file system named `/proc` that lets you see the memory where they are stored. ] --- name: listing_dirs ## Listing Directory Contents - The `ls` command displays the contents of one or more directories. To see the contents of the current directory, just type `ls`: ```bash [sweiss@cslab12 cs395.86]$ ls .: bin data demos scratch temp [sweiss@cslab12 cs395.86]$ ls . ``` - The `ls` command is covered later. --- name: filenames ## About File names - If you are new to Unix, you need to know some important facts: - File names are case sensitive: `temp` and `Temp` are different names. - File names that begin with a period `.` are not displayed by `ls` and therefore act like "hidden" files - File names should not contain spaces: use underscores to separate words, as in `school_records` or `source_downloads`. - File names can contain most other characters, but convention is to use `_`, `-`, `.`, and letters and digits (for good reasons.) - In the shell, file name extensions are meaningless for the most part. Some programs, such as compilers, use them, but shell commands ignore them. --- name: pathnames ## Relative Pathnames - In addition to absolute pathnames, all files and directories can be referred to by .greenbold[relative pathnames]. - Whereas absolute pathnames always start with the slash `/`, a .bluebold[relative pathname] is a path that is relative to the current working directory .greenbold[and does not start with `/`]. - In addition to the period, which is the current working directory, all shells define the special notation consisting of two periods `..` to represent the .greenbold[parent directory] of the current directory, which is the directory that contains it. - Example - The pathname `./scratch/README` is a relative path from ```bash /data/biocs/b/student.accounts/cs395.86 ``` to the file whose absolute pathname is ```bash /data/biocs/b/student.accounts/cs395.86/scratch/README ``` --- name: navigating_dirs ## Navigating Directories - The `cd` command (think " .redbold[c]hange .redbold[d]irectory " ) changes your current directory. You give it the name of a directory, either as an absolute pathname or as a relative pathname. - .redbold[Examples]. ```bash $ pwd /data/biocs/b/cshome/sweiss $ cd /data/biocs/b/student.accounts/cs395.86/ $ pwd /data/biocs/b/student.accounts/cs395.86 $ cd ~/hunter $ pwd /data/biocs/b/cshome/sweiss/hunter $ cd ../../cs395.86 $ pwd /data/biocs/b/student.accounts/cs395.86 ``` --- name: command_types ## Types of Commands - Every Unix command can be any of the following: 1. an executable file (either binary or in a scripting language) 1. a command that is built into the shell 1. an .greenbold[alias] 1. a .greenbold[shell script] that is in your .greenbold[environment] Sometimes they can be more than one of these. - .redbold[Examples] - Commands like `ls` and `who` are executable binaries. `ls` is the file `/bin/ls` and `who` is the file `/usr/bin/who`. - Commands like `cd` and `bg` are usually .bluebold[shell built-ins], which are commands that are implemented within the shell program itself. - Commands like `pwd` and `echo` are .greenbold[both shell built-ins and executable files]. --- name: learning_commands ## Learning About Commands - The `type` command (which is a shell built-in) displays the type(s) of its arguments: ```bash $ type pwd cal pwd is a shell builtin cal is /usr/bin/cal $ type type type is a shell builtin $ type rm rm is aliased to `rm -i' ``` Although `type` says that `rm` is an alias, it is also the executable `/bin/rm`. - If a command is an executable file, the `which` command will display the absolute pathname of the executable file that you will run when you type its name.super[1]: ```bash $ which ls /bin/ls ``` The `which` command only displays output for commands that are files, not built-ins: it ```bash $ which bg $ ``` .footnote[ 1 Sometimes a command has multiple versions. This shows the one you will use. ] --- name: man_pages ## Getting Help: The Manual Pages - These days many people start looking for help on using a particular command by doing a search in their browser, but there are resources found right on your machine to get help while you are working, .greenbold[_and they do not require an internet connection_]. - If you know the name of the command but do not know how to use it, the `man` command is the first thing to try. `man` is short for .redbold[man]ual. - To access the __manual page__ for a command, type `man` followed by the command name. For example, the "man page" for `ls` is displayed by ```bash $ man ls ``` - To get a one-line summary of every version of a command use `whatis`: ```bash $ whatis who who (1) - show who is logged on who (1posix) - display who is on the system ``` --- name: man_page_searching ## Searching Through Man Pages - If you do not know the name of a command, there are two types of keyword searches you can try: `man -k <search term>` and `apropos <search term>`. The output below is truncated. ```bash $ man -k "text editor" ed (1) - line-oriented text editor ex (1) - Vi IMproved, a programmer's text editor gedit (1) - text editor for the GNOME Desktop vi (1) - Vi IMproved, a programmer's text editor vim (1) - Vi IMproved, a programmer's text editor xedit (1) - simple text editor for X ``` - Usually the volume of results is very large and you will want to .greenbold[filter] them. .greenbold[Filters] are covered later, but one simple thing you can do is to .greenbold[pipe the output through `more`].super[1]: ```bash $ apropos filter | more apgbfm (1) - APG Bloom filter management program bcomps (1) - biconnected components filter for graphs bf_compact (1) - shell script to compact a bogofilter directory bf_compact-bdb (1) - shell script to compact a bogofilter directory bf_copy (1) - shell script to copy a bogofilter working directory bf_copy-bdb (1) - shell script to copy a bogofilter working directory bf_tar (1) - shell script to write a tar file of a bogofilter directory to stdout ... ``` .footnote[ 1 To learn more about pipes, go to [Pipes](#pipes) ] --- name: info ## The Info System - The GNU Project provides an alternative to the man pages for their programs, called the .bluebold[info system]. Only GNU commands have an `info` page. Type ```bash $ info ``` to see the starting page for the `info` system, which lists its table of contents. - The `info` pages are .greenbold[hyper-linked into a tree-like structure]. Pages that describe commands are called .bluebold[nodes]. The navigation system uses commands to move around the nodes, such as `p` for previous node, `n` for the next node, and so on. - To learn how to use it, read the very first page that is brought up with the `info` command: ```bash File: dir, Node: Top, This is the top of the INFO tree. This is the Info main menu (aka directory node). A few useful Info commands: 'q' quits; 'H' lists all Info commands; 'h' starts the Info tutorial; 'mTexinfo RET' visits the Texinfo manual, etc. ... ``` --- name: bash_help ## Help for Shell Builtins - Man pages do not provide help for using shell built-ins, unless you are willing to read the 4356-line man page for `bash` itself and search for the specific help on that builtin. - But `bash` has a (builtin) command named `help` that provides help on all shell built-ins. You do not need to give it the exact name - you can give it the first few letters and it will display help for all commands starting with those letters: ```bash $ help help help: help [-dms] [pattern ...] Display information about builtin commands. Displays brief summaries of builtin commands. If PATTERN is specified, gives detailed help on all commands matching PATTERN, otherwise the list of help topics is printed. Options: -d output short description for each topic -m display usage in pseudo-manpage format -s output only a short usage synopsis for each topic matching PATTERN Arguments: PATTERN Pattern specifying a help topic Exit Status: Returns success unless PATTERN is not found or an invalid option is given. ``` --- name: command_categories ## Categories of Commands - There are thousands of commands in a typical Linux system. This slide and the next one list categories with the most useful commands in each. Some commands appear in multiple categories. This list generally excludes utility programs such as compilers and other programming tools. - File and Directory Processing: - `basename, cat, cd, chgrp, chmod, cmp, cp, diff, dirname, file,` `find, less, ln, ls, mkdir, more, mv, objdump, od, pwd, realpath,` `rename, rm, rmdir, stat, strings, touch, view` - Filters - `awk, cat, cut, grep, egrep, fold, fgrep, head, join, less, more, od, paste, sed, shuf, sort,` `split, tac, tail, tee, tr, uniq, wc` - System and User Information - `date, domainname, du, free, groups, hostname, id, last, printenv, uptime,` ` users, times, which, who, whoami, w` --- name: misc_commands ## More Commands - Process and Job Control - `bg, fg, jobs, kill, pkill, pgrep, ps, script, timeout, ^C, ^Z, ^D, ^S, ^Q` - These commands are hard to categorize but either important or useful: - `bc, echo, false, help, info, man, seq, shuf, sleep, stty, true, tty, yes` --- name: cmmd_syntax ## Command Syntax - In general, commands may have options and arguments. The general form is ```bash commandname [options] arguments ``` where square brackets `[ ]` mean that the thing enclosed in them is optional. - .redbold[Example]. The man page for the `which` command shows the synopsis ```bash SYNOPSIS which [-a] filename ... ``` In this case there is just a single optional `-a` and the command allows multiple filename arguments, indicated by the ellipsis, `...`. - Options begin with a hyphen `-`. Multiple options are usually allowed. For most commands, if the option does not require an argument of its own, it may be combined with other options using a single hyphen. --- name: cmmd_syntax2 ## Simple Command Syntax - Commands can have single-letter options, options that are words, and options that have required arguments. The `zip` command, which creates compressed archives, is a good example. (Part of the synopsis is removed to simplify this.) ```bash SYNOPSIS zip [-aABcdDeEfFghjklLmoqrRSTuvVwXyz!@$] [--longoption ...] [-b path] [-n suffixes] [-t date] [-tt date] [zipfile [file ...]] ``` - All single-letter options can be listed as a single string preceded by a hyphen or separately in any order: ```bash $ zip -rcC myzipfile mydirectory ``` ```bash $ zip -r -c -C myzipfile mydirectory ``` - Some options require arguments. The synopsis indicates this by listing it separately. The `-t` option for `zip`, for example, requires a date. ```bash [-t date] ``` The man page shows what forms a date can be. --- name: cmmd_syntax3 ## Complex Command Syntax - When some options require arguments and others do not, the ones with arguments must be specified separately. It usually does not matter what order the options are, though some man pages will state otherwise. Both of these do the same thing: ```bash $ zip -rt 1991-12-07 infamy foo $ zip t 1991-12-07 -r infamy foo ``` They both add all the files in `foo` and its subdirectories that were last modified on or after December 7, 1991, to the zip archive `infamy.zip`. But this command cannot be written like this: ```bash $ zip -tr 1991-12-07 infamy foo ``` --- name: misc_commands ## Miscellaneous Commands - Among the commands that do not neatly fit into a particular category, I included `echo` and `seq`. These are both very useful and will be used for several examples. - The `echo` command displays the words that follow it on the command line, as in ```bash $ echo "Dan Quaye once said, 'The future will be better tomorrow.'" Dan Quaye once said, 'The future will be better tomorrow.' ``` It is useful because `bash` will do .greenbold[variable evaluation] before giving the words to `echo`, as in ```bash echo "My screen has $COLUMNS columns right now." My screen has 80 columns right now. ``` --- name: misc_commands2 ## Miscellaneous Commands - The `seq` command generates and prints sequences of numbers. Its general form is ```bash seq first increment last ``` If first or increment are omitted, they default to 1. The separator is a newline, but the -s option can be used to give it a different separator: ```bash $ seq -s ' ' 1 10 1 2 3 4 5 6 7 8 9 10 ``` or ```bash $ seq -s '-' 10 2 20 10-12-14-16-18-20 ``` or ```bash $ seq -s"--" 20 1--2--3--4--5--6--7--8--9--10--11--12--13--14--15--16--17--18--19--20 ``` --- name: redirection_examples1 ## The Power of I/O Redirection - One of the brilliant inventions of Unix is .greenbold[I/O redirection]. To get just a small taste of its power, run a few of these commands, all of which use the .greenbold[pipe] redirection operator: 1. Generate a random string of length 32 (suitable for a strong password): ```bash $ cat /dev/urandom | tr -dc 'a-zA-Z _&%$#0-9-' | fold -w 32 | head -1 ``` If you want to store it in a new file named `mypasswd`, then use ```bash $ cat /dev/urandom | tr -dc 'a-zA-Z _&%$#0-9-' | fold -w 32 | head -1 > mypasswd ``` 1. Generate a sorted list of user names of people who have logged into the system recently (excluding `root` and `wtmp`): ```bash $ last -w | sort -k1 | awk '{print $1}' | uniq | grep -Ev 'wtmp|root' ``` --- name: redirection_examples2 ## More Examples of Redirection 1. Suppose you have a large directory of source files and want a list of every line in which a call to the function `read` is made. Knowing it has three arguments we can use this: ```bash $ grep -Ron 'read(.*,.*,.*)' * | awk -F: '{printf "%-32s %6d %s\n",$1,$2,$3}' ``` A portion of output is below: ``` chapter05/restart_demo.c 86 read(0, &c, 4) chapter05/canon_mode_test2.c 170 read(0, input, 10) chapter05/nonblockdemo.c 186 read(STDIN_FILENO, &ch, 1) ) > 0 ) chapter05/simplevi.c 394 read(STDIN_FILENO, &c, 1) > 0 ) chapter05/simplevi.c 410 read(STDIN_FILENO, &c, 1) > 0 ) chapter05/simplevi.c 577 read(STDIN_FILENO, &c, 1) chapter05/simplevi.c 607 read(STDIN_FILENO, &c, 1) chapter05/simplevi.c 609 read(STDIN_FILENO, &c, 1) chapter08/upcased2.c 159 read( publicfifo, (char*) &msg, sizeof(msg)) > 0 ) chapter08/upcased2.c 189 read(client_rawtext_fd, buffer, PIPE_BUF)) > 0 ) ``` --- name: redirection_examples3 ## More Examples of I/O Redirection 1. This one uses `awk` and `sort` together. New York State maintains an open data set of Covid testing by date and county. The following command prints a line for each date showing the number of new positive test cases in the given county, sorted by increasing number of cases: ```bash $ awk -F, '($2 == "Queens")\ { printf "New positive Covid cases in Queens on %s is %d\n",$1,$3}' \ New_York_State_Statewide_COVID-19_Testing.csv | sort -n -k10 ``` - Notice how these examples rely heavily on the power of `awk`. `awk` is a very powerful filter that is worth learning. --- name: streams ## Streams, Files, and Redirection - A bluebold[stream] is a flow of bytes into or out of a running process. In Unix, a stream is implemented with a data structure that includes buffers to store the bytes and various data members to control its flow and keep track of the status of the stream. - Unix provides every running process with three .greenbold[standard streams]: - .greenbold[standard input], .greenbold[standard output], and .greenbold[standard error] <img src="img/stdinout.GIF" width="300" style="float:right;" > - Streams can be connected to devices or files or other processes, as you will see shortly. - When a process is created, it is given an array that has pointers to its open files; pointers to these three streams occupy the first three array entries, namely standard input is index 0, standard output 1, and standard error 2. - Initially, standard input is connected to the keyboard (terminal device), and standard output and error, the terminal device*. .footnote[ * The terminal window is called a pseudo-terminal in Unix. ] --- name: file_descriptors ## File Descriptors and Streams - The index values in the array of open files belonging to the process are called .bluebold[file descriptors]. The table below summarizes the different views of the standard streams. File<br> Descriptor | Stream | Associated Device | C symbolic name :---|:---|:---|:--- 0 | Standard Input | Keyboard |stdin 1 | Standard Output | Screen or Terminal Window | stdout 2 | Standard Error | Screen or Terminal Window |stderr --- name: io_redirection ## I/O Redirection - All shells allow for any stream to be disconnected from its default device and reconnected to or from files, the streams of other processes, or other devices. This is called .bluebold[I/O redirection]. - Attaching a file to the standard input is called .bluebold[input redirection]. The `<` <!-- > --> operator is the .bluebold[input redirection operator]: ```bash command < file ``` <!-- > --> Example: ```bash cat < file1 ``` <!-- > --> The contents of file1 are redirected to the standard input of the `cat` command, which is not very useful, but illustrates the idea. --- name: input_redirection ## Input Redirection - The `bc` command can be used as a simple calculator. If you type `bc` on the command line, it waits for you to enter an arithmetic expression such as 25 + 4 or something more complex. When you type <enter> it evaluates and prints the expression. `6 + 4` -- <br> `10` -- <br> `10 ^ 3` -- <br> `1000` -- <br> `print "The product of 10 and 200 is " , 10 * 200, "\n"` -- <br> `The product of 10 and 200 is 2000` -- - Suppose a file named `expressions` contains the following lines: ```bash print "10^2 = ", 10^2, "\n" print "2^10 = ", 2^10, "\n" ``` -- - Running the command `bc < expressions` <!-- > --> at the command prompt: ```bash $ bc < expressions 10^2 = 100 2^10 = 1024 ``` <!-- > --> shows that `expressions` replaced `bc`'s standard input. --- name: output_redirection ## Output Redirection - Attaching a file to the standard output is called .bluebold[output redirection]. The `>` operator is the .bluebold[output redirection operator]: ```bash command > file ``` Example ```bash cat file1 file2 file3 > combined_file ``` concatenates files `file1`, `file2`, and `file3` into a new file named `combined_file`. If `combined_file` already existed and the `bash` variable `noclobber` is set,q the command fails and you will see an error message such as ```bash bash: combined_file: cannot overwrite existing file ``` -- To overcome this, use the `>|` operator, which will forcibly replace the existing file: ```bash cat file1 file2 file3 >| combined_file ``` - --- name: error_redirection ## Error Redirection - Attaching a file to the standard error is called .bluebold[error redirection]. The `>` operator is also used to redirect standard error, but with a slight modification; use ```bash command 2> file ``` Note: .redbold[No space] between `2` and `>`. `2` is the file descriptor for the standard error stream. In general, if _n_ is a file descriptor, _n_`>` redirects the stream associated with it to the file. -- - To send the errors of a command to one file and the standard output to another, the simplest solution (there are a few ways to do this) is: ```bash command 2> error_file > output_file ``` If either file exists and `noclobber` is set, it will fail; to overwrite them use ```bash command 2>| error_file >| output_file ``` - To redirect the standard output and the standard error to the __same__ file, use ```bash command &> file ``` --- name: redirection_examples ## Redirection Examples - Some commands can produce many error messages. Sometimes you don't care about the errors. If you try to display the _entire file system_ (bad idea) using `ls -R` there will be many `permission denied` errors. The way to discard them is like this: ```bash ls -R / 2>/dev/null > verybigfile ``` .redbold[Lesson]: `/dev/null` is a black hole; any data written to it is discarded. - We can take advantage of the fact that the `cat` command reads from standard input if it is not given a filename as an argument_, to create a plain text file while taking notes, like this: ```bash $ cat > notes I am typing on the keyboard and trying to write down this stuff before I forget. ^D (Control D to stop) $ cat notes I am typing on the keyboard and trying to write down this stuff before I forget. ``` We must type Control-D to terminate the `cat` command and close the new file named notes. --- name: appending ## Appending - To append to a file (or to create it if it does not exist) use the `append redirection operator` `>>`. ```bash command >> file ``` which adds the output of the command to the end of the file. - Sometimes you will find it useful to log the results of a command that you run, or perhaps log its errors. Suppose `backup` is some backup command that you run every day. You can do either: ```bash backup 2>> ~/.backup_errlog >> ~/.backuplog ``` which sends the errors of the `backup` command to the `.backup_errlog` log file and the standard output to the `.backuplog` file in the home directory. - Here is a `bash` loop that stores the output of the `git log` command for a set or three repositories in a single file named `logfile`. `bash` loops are covered elsewhere. ```bash $ for i in repo1 repo2 repo3 ; do echo -e "Repository $i log:\n\n" >> logfile cd $i; git log >> ../logfile; cd .. ; done ``` --- name: pipes ## Pipes - Attaching the standard output of one command to the standard input of another is done by creating a __pipe__. The `|` operator is the __pipe operator__. - Syntax: ```bash command1 | command2 ``` - .redbold[Example]. This pipeline prints the number of lines of each C source code file in a given directory, sorted by filename. ```bash $ wc -l *.c | sort -k2 | awk '{printf "%s:\t%d\n",$2,$1}' ``` - The real power of pipes is that they can be used with __filters__. More examples of pipes will follow the introduction of filters. --- name: viewing_files ## Viewing Files - Many of the commands to view the contents of files are simple to use. Some are intended for text files only, others any type of file. Because they are easy, no examples are given. Look up the syntax and options in the man pages. - All are used by following the command name with a list of files. They include: - `more` and `less` - display a screenful at a time of the given text files - `view` - brings up the `vi` editor in read-only mode. Use it only if you know `vi`. It can be given multiple text file names. - `cat` - "__c__oncatenates __a__nd __p__rints" its list of text file arguments. It whizzes by on the screen, so use it only to see small files. - `od` - displays an "octal dump" of any file, including binaries. It has lots of options to make it very useful. See also `objdump`. - `objdump` - displays information about one or more object files. The options control what particular information to display. - `strings` - print the strings of printable characters in files. It is very useful for finding the strings in compiled code. --- name: ls ## `ls` - This lists information about files and directories - Syntax: ```bash ls [option]... [file]... ``` If no files are given it lists information about the current directory. - Some very useful options: ```bash -l - produce a "long" listing -a - show all files, including hidden ones -F - append trailing character to classify type -t - sort by time of last modification, most recent first -R - recursively list subdirectories ``` - Examples - list all files in current directory with most recent first, classifying ```bash $ls -atlF ``` - list all files sorted by access time ```bash $ls -atulF ``` --- name: basename ## `basename` - `basename` strips leading directory path and suffix from filenames - Syntax: ```bash basename pathname [suffix] ``` outputs `pathname` with the leading path removed and, if it has a suffix matching `suffix` with that removed also. - Examples: ```bash basename /usr/bin/sort -> sort basename include/stdio.h .h -> stdio basename include/stdio.h .foo -> stdio.h # i.e., suffix does not match basename ~/hunter/cs395.86_s19/blogs/stewart-weekly -weekly -> stewart ``` --- name: dirname ## `dirname` - `dirname` strips the filename from a pathname, leaving the directory path. - Syntax: ```bash dirname pathname [pathname] ... ``` outputs each pathname with its last non-slash component and trailing slashes removed. - Examples: ```bash dirname /usr/bin/sort -> /usr/bin dirname /usr/bin/gcc /usr/lib/gcc /usr/share/man/man1/gcc.1.gz -> /usr/bin /usr/lib /usr/share/man/man1 dirname ~/hunter/cs395.86_s19/blogs/stewart-weekly -> /home/stewart/hunter/cs395.86_s19/blogs ``` --- name: find ## `find` - Its man page states, "search for files in a directory hierarchy", but this is an understatement. It is one of the most powerful commands available in Unix. - The `find` command allows you to apply commands and actions to all files matching a set of search criteria in one or more subtrees of the Unix file system. - Simplified syntax (some options suppressed): ```bash find [starting-point...] [expression] ``` - `starting-points` are the directories to act as the roots of the hierarchies to search. - `expression` describes what is to be searched for; it includes search criteria as well as actions to perform. - With no expression, `find` displays every file in the trees rooted at the `starting-points`. The default action is `-print` ```bash find . -print ``` is equivalent to ```bash find . ``` --- name: find_by_name ## finding by file name - Finding (printing paths to) all files in the directory `dir` whose name ends in `.cpp`: ```bash find dir -name "*.cpp" ``` .redbold[Lesson]: The `*` is a shell wildcard that matches 0 or more characters, including the period if it is the first character in the name. - Finding all files in `dir` whose name is exactly `main.cpp` ```bash find dir -name "main.cpp" ``` -- - Finding all files in `dir` whose name ends in any of `.jpg`, `.JPG`, `.JPg`, etc: ```bash find dir -iname "*.jpg" ``` .redbold[Lesson]: `-iname` is a case-insensitive version of `-name`. -- - Finding all files in `dir` whose name ends in any of `.jpg`, `.JPG`, `.jpeg`, `.JPEG`, etc: ```bash find dir -iname "*.jpg" -o -iname "*.jpeg" ``` .redbold[Lesson]: Expressions return `true` or `false`. `-iname` is a __test__ applied to each file as it is found. If the filename matches, it returns `true` otherwise `false`. The `-o` is a logical OR-operator; its operands above are `-iname "*.jpg"` and `-iname "*.jpeg"`. If either is true then the filename passes the test. --- name: find_by_time ## finding by time stamp - Unix timestamps files with three stamps: time of last access, time of last modification, and time of last change of status (file properties). - Finding files in `dir` that have been __modified__ within the past 3 hours: ```bash find dir -mmin -180 ``` .redbold[Lesson]: `-mmin` expects an argument in minutes. The `-` in front of 180 means "less than". -- - Finding files in `dir` that have been modified more than 3 hours ago: ```bash find dir -mmin +180 ``` -- - Finding files in `dir` that have been __accessed__ (use `-amin`) exactly 3 hours ago: ```bash find dir -amin 180 ``` -- - Finding files in `dir` whose __status changed__ (use `-cmin`) within the past 8 hours: ```bash find dir -cmin -480 ``` --- name: find_misc ## finding by other properties - In Unix, properties such as size, type, permissions, user ownership, group ownership, and more, are stored in a special structure called an __inode__. `find` can test any of these properties. - Finding files in `dir` larger than 500 Kilobytes (1024 bytes): ```bash find dir -size +500k ``` -- The letter after the number can be `c`, `w`, `k`, `M`, or `G`. Guess what they stand for. - Finding files in `dir` that are executable: ```bash find dir -executable ``` -- - Finding files in `dir` that are owned by user stewart and group cs_ossd ```bash find dir -user stewart -a -group cs_ossd ``` .redbold[Lesson]: `-a` is the logical AND-operator --- name: find_misc2 ## finding by other properties - Finding files in `dir` for which people other than the owner and the group have write access, whether or not the owner or group does. ```bash find dir -perm -002 ``` It is dangerous to let anyone be able to write to a file. This looks for all such files. - Finding files in `dir` for which the owner has read, write, execute permission and no one else has access of any kind: ```bash find dir -perm 700 ``` - Finding files in `dir` that are __regular__ files: ```bash find dir -type f ``` To find directories, replace `f` with `d`. --- name: find_actions ## taking actions when files are found - You can use add __actions__ to expressions. These actions can be applied to the files for which the test returns true, or to a set of arguments that follow the action. - Useful actions include `-print`, `-prune`, and `-exec`. There are many others. - `-prune` is used to prune the search, i.e., prevent it from descending the tree. - `-exec <command>` executes the <command> that follows it. - Run the `file` command on every regular file below the current directory : ```bash find . -type f -exec file '{}' \; ``` .redbold[Lesson]: `exec` is followed by a command. `{}` after the command is replaced by the file that matched the test. It must be written in quotes, and the semicolon must be escaped with backslash as shown. - Remove every file whose names ends in `~` below the current directory (dangerous if you make a mistake): ```bash find . -name "*~" -execdir /bin/rm '{}' \; ``` --- name: filter_intro ## Filter Introduction - A filter is a UNIX command whose input and output are plain text, and that expects its input from standard input and puts its output on standard output. - Most filters will also read their input from one or more filenames listed on their command-line. - Filters are useful because their output is a transformation of their input, in one way or another, such as by sorting it, removing words or lines based on a pattern or on their position in the line or file. - Filters are chained together with multiple pipes to do become the workhorses of Unix systems. --- name: filter_synopsis ## Filter Synopsis Filter | Description :--- |:--- `awk` | pattern scanning and processing language `cat` | concatenate files and print on the standard output `cut` | remove sections from each line of files `fold` | wrap each input line to fit in specified width `grep, egrep, fgrep` | print lines matching a pattern `head` | output the first part of files `join` | join lines of two files on a common field `less` | does more than `more`; see `more` below. `more` | file perusal filter for crt viewing `od` | dump files in octal and other formats `paste` | merge lines of files `sed` | stream editor for filtering and transforming text `shuf` | generate random permutations `sort` | sort lines of text files `split` | split a file into pieces `tac` | concatenate and print in reverse order `tail` | output the last part of files `tr` | translate or delete characters `uniq` | report or omit repeated lines `wc` | print newline, word, and byte counts for each file --- name: cat_tac ## `cat` and `tac` - The `cat` command is a technically a filter, but without options it does no transformation: its output is exactly its input. - It does have some handy uses. The `-n` option numbers lines, `-b` numbers non-blank lines, `-s` _squeezes_ blank lines, and `-v` shows non-printing characters: ```bash ls . | cat -n 1 bash_tutorial_01.html 2 css/ 3 img/ 4 js/ ``` lists the current directory and numbers the lines. - `tac` prints the lines in reverse order. I rarely have use for it. ```bash ls . | cat -n | tac 4 js/ 3 img/ 2 css/ 1 bash_tutorial_01.html ``` --- name: head_tail ## `head` and `tail` - Simply put, `head` displays the first _N_ lines of its input and `tail`, the last _N_ lines. By default for both, _N_ is 10. To print a different number of lines, explicitly use `-N` where _N_ is a positive integer: ```bash head -1 myfile ``` displays just the first line of `myfile`, and ```bash tail -1 myfile ``` displays the last line. -- - One way to print the _nth_ line of a file is like this: ```bash head -4 myfile | tail -1 ``` which prints the 4th line of `myfile`. So I can use the following pipeline to get the summary of any command from the man page for it: ```bash man awk | head -4 | tail -1 ``` since the summary is always line 4. --- name: sorting ## `sort` - `sort` is one of the most useful and easy to use filters: ```bash sort myfile ``` will sort the text file named `myfile` and print it on standard output. -- - But its exact behavior varies from one system to another; there are many different implementations of `sort`. -- - By default it uses the current `locale` settings (the _collating order_ set in your environment*). The major difference in English is whether uppercase precedes lowercase or whether case is ignored. - Locales are a subject way ahead of us. Suffice it to say that locale environment variables control how information is displayed in the terminal, such as the language and character set, numbers, dates, times, and more. .footnote[ * The collating order is the order of the characters in the character code of the terminal, which is usually ASCII or UTF-8. ] --- name: sort_examples1 ## `sort` Examples (1) - Assume `myfile` contains the lines ```bash a b A B ``` Forcing the locale to be U.S. English and running `sort`: ```bash LC_COLLATE="en_US.UTF-8" sort myfile a A b B ``` Forcing the locale to be the "C" locale and running `sort`: ```bash LC_COLLATE=C sort myfile A B a b ``` --- name: sort_examples2 ## `sort` Examples (2) - To force `sort` to ignore case and the current locale, and fold upper and lower case together, using GNU's `sort`, the option is `-f`: ```bash LC_COLLATE=C sort -f myfile a A b B ``` - `sort` by default will treat numbers like strings. For example, it will sort 1, 2, 10, 20 in this order: 1, 10, 2, 20. - To tell `sort` to sort .bluebold[numerically], use `-n`. - To tell `sort` to .bluebold[reverse] its order, use `-r`. - To tell `sort` to delete duplicate lines on output, use `-u`. - `sort` can sort by __fields__ in a line. The GNU version uses the `-k` option to specify the specific _key_ position (1-based). The `-t` option tells `sort` what character keys in the line. By default `sort` uses whitespace. If a file has colon-separated fields, and you want to sort numerically by field 2, use ```bash sort -t':' -k2 -n myfile ``` --- name: uniq ## `uniq` - `uniq` filters out matching adjacent lines from its input stream, sending unique lines to output. If the input stream is sorted and has duplicates, this produces output with duplicates removed: ```bash sort mydata | uniq ``` produces a stream of unique lines from the file `mydata`. - It has several useful options as well: ```bash -i - ignore case when trying to match -c - prefix lines by the number of occurrences -d - only print duplicate lines, one for each group -u - only print the unique lines ``` --- name: combining_filters ## Combining Filters - Putting some of this filters together, NYC Open Data has a database of baby names whose lines look like this: ```bash 2011,FEMALE,HISPANIC,Geraldine,13,75 2011,FEMALE,HISPANIC,GIA,21,67 2011,FEMALE,HISPANIC,GIANNA,49,42 ... ``` - Many names are repeated. We can get the frequencies of each name using this pipeline, assuming the file is named `babynames.csv`: ```bash sort -k4 babynames.csv | cut -d, -f4 | uniq -ic ``` where `sort` sorts using the 4th field, after which `cut`.super[1] prints only the 4th field, then `uniq` filters out duplicates and puts counts to the left of the names, ignoring case. We can get the ten most frequent names like this: ```bash sort -k4 babynames.csv | cut -d, -f4 | uniq -ic | sort -k1 -nr | head -10 ``` .footnote[ 1 The `cut` filter is described later in these slides. ] --- name: grep_family ## Filters: `grep`, `egrep`, and `fgrep` - These commands are the most powerful of all. `grep` and `egrep` are given a pattern, called a __regular expression__, and use this to filter lines that match the pattern. `fgrep` is a _fast_, fixed-string version that does not use patterns. - Regular expressions are complex; mastering them is worth the effort because they are used by `vi`, `sed`, `ed`, `awk`, `grep`, and `egrep`. - Some examples to start: ```bash grep '\<cout\>' prog.cpp ``` -- prints all lines containing the exact word `cout` in file `prog.cpp`. The brackets `\<` and `\>` mean "match only .redbold[whole words]" so this does not match "scout" or "couty". -- ```bash grep -c '^$' prog.cpp ``` -- prints a count of the number of empty lines in `prog.cpp`. -- ```bash grep '\/\*.*\*\/' prog.cpp ``` -- prints all lines in `prog.cpp` that have C-style comments `/* ... */`. - .redbold[IMPORTANT. Always enclose the pattern in single quotes.] --- name: regex1 ## Regular Expression Rules - In the rules that follow, `\0` denotes an empty string. - The complete set of rules can be found in the __regex__ man page in section 7, which defines the POSIX-compliant regular expressions. - Any sequence of characters matches itself: __abc__ matches the string "abc". - A regular expression followed by \* matches the concatenation of 0 or more strings each of which is matched by the regular expression. \* is called the __closure__ operator. - __`a*`__ matches 0 or more `a`'s: - `\0`, `a`, `aa`, `aaa`, ... - __`ab*`__ matches `a` followed by 0 or more `b`'s: - `a`, `ab`, `abb`, `abbb`, ... - __`ab*ac*`__ matches `a` followed by 0 or more `b`'s followed by `a` followed by 0 or more `c`'s: - `aa`, `aba`, `aac`, `abba`, `abac`, `aacc`, `abbba`, `abbac`, ... --- name: regex2 ## Regular Expression Rules (2) - Use __`\( \)`__ to group for applying \* to more than one character: - __`\(ab\)*`__ matches 0 or more `ab`'s: - `\0`, `ab`, `abab`, `ababab`, ... - If you use __extended regular expressions__ either by writing `grep -E` or by using `egrep` instead, you can use `+`, the __positive closure__ operator. It matches __1 or more__ strings each of which is matched by the preceding regular expression. You can also use ordinary parentheses for grouping: ```bash egrep '(ab)+' myfile grep -E '(ab)+' myfile ``` both match all lines in `myfile` that have 1 or more consecutive `ab` substrings. - Warning: ```bash grep '(ab)*' myfile ``` only matches `\0`, `(ab)`, `(ab)(ab)`, ... because `grep` was used instead of `egrep`. --- name: regex3 ## Regular Expression Character Classes - The period __`.`__ matches any single character. <!-- [ [ --> | | |:--- |:--- | __`[list-of-characters]`__ | matches any single character in the list. | __`[a6j&]`__ | matches a, 6, j, or & | __`[0-9]`__ | matches any single digit | __`[a-zA-Z0-9]`__ | matches any letter or digit |__`[]]`__ | matches right square bracket `]` |__`[0-9-]`__ |matches any single digit or hyphen |__`[-0-9]`__ | matches any single digit or hyphen |__`[_a-zA-Z0-9]`__ | matches any letter, digit or underscore. - There are shorthands for these. For example, __`\w`__ is a shorthand for __`[_a-zA-Z0-9]`__. These characters are called __word characters__. - The `^` inside brackets means the complement: ` ` | ` ` :--- |:--- __`[^a6j&]`__ | matches anything except a, 6, j, and & __`[^0-9]`__ | matches anything except a digit. --- name: regex4 ## Regular Expression Character Classes (2) - You can combine character classes with the * operator to create useful patterns: ` ` | ` ` :--- |:--- __`\(c[acgt]g\)*`__ | matches 0 or more sequences of `cag`, `ccg`, `cgg`,or `ctg` __`[1-9][0-9]*`__ | matches any decimal numeral except 0 __`[A-Z][a-z]*`__ | matches words that start with an uppercase letter. __`[a-zA-Z_][_a-zA-Z0-9]*`__ | matches C/C++ identifiers. __`\(...\)*`__ | matches any string whose length is a multiple of 3. --- name: anchors ## Regular Expressions: Anchors - The caret __`^`__ anchors a regex to the beginning of a line, and the dollar sign, __`$`__, anchors it to the end of the line. - __`\<`__ anchors a regex to the start of a word. This means that the character before it must be a non-word character. - __`\>`__ anchors the regex to the end of a word. This means that the character after it is a non-word character. ` ` | ` ` :--- |:--- __`^drwx`__ | matches lines whose first 4 characters are drwx __`^\w`__ | matches lines that begins with a letter or digit or underscore __`abcd$`__ | matches lines whose last 4 characters are abcd __`^abc$`__ | matches lines that contain only abc __`^$`__ | matches empty lines __`^[ ]*$`__ | matches empty lines or lines containing only spaces __`\<fred`__ | matches any word starting with `fred` but not words like `alfred` __`fred\>`__ | matches any word ending with `fred` but not words like `freddy` __`\<fred\>`__ | matches exactly `fred` surrounding by non-word characters --- name: backreferences ## Regular Expressions: Backreferences - When you enclose a basic regular expression in __`\( \)`__ brackets, or an extended regular expression in ordinary parentheses __`( )`__, the string that matched it is "remembered" for future use. - The regular expression __backreference__, __`\1`__, matches the first such "remembered string." In __`\(aa*\)b\1`__ any string that matches `aa*` is saved into a storage cell named \1. The only strings that this expression matches are `aba`, `aabaa`, `aaabaaa`, `aaaabaaaa`, ...,. - In general, the expressions `\1`, `\2`, `\3`, …, `\9` remember matches of the 1st, 2nd, 3rd, up to 9th parenthesized regular expressions. - The expression __`^\(.*\):\(.*\)::\1:\2$`__ matches lines of the form `x:y::x:y` where `x` and `y` are possible empty strings, such as abc:666::abc:666 --- name: sed1 ## Filters: `sed` - Knowing something about regular expressions will make it easier to see the power of `sed`, the Unix stream editor. From the `sed` man page: - A .bluebold[stream editor] is used to perform basic text transformations on an input stream (a file or input from a pipeline). While in some ways similar to an editor which permits scripted edits, `sed` works by making only .redbold[one pass] over the input(s), and is consequently more efficient. But it is `sed`'s ability to filter text in a pipeline which particularly distinguishes it from other types of editors. - These few slides are designed to show simple ways to use `sed`, mostly by example, but also by explaining general rules. -- - The usual way to invoke `sed` is to give it a .greenbold[sed-script], enclosed in single quotes, either followed by the files on which you want to run it, or on the right-hand side of a pipe: ```bash sed 'sed-script' file1 file2 file3 sort file | sed 'sed-script' ``` - A sed-script is usually a single command but it can be multiple commands as well. --- name: sed2 ## `sed` Substitution Command - .redbold[Example]. The substitution command is the letter `s`. It expects a pattern to match, called the .greenbold[target], and a replacement pattern, like this: .greenbold[`s/target/replacement/`]: ```bash $ echo -e "Hello World\nHello World\nHello World" | sed 's/World/Planet Earth/' Hello Planet Earth Hello Planet Earth Hello Planet Earth ``` In `sed` by default, the command is applied to every input line, so this command replaces "World" by " Planet Earth" on every line. - Commands can be given .greenbold[address-specifiers], which are ways to specify one or more lines, and then `sed` applies the commands only to the specified lines. - The address specifier comes .redbold[before] the command. .redbold[Examples]: ```bash sed '1s/World/Planet Earth/' # replace only on line 1 sed '2s/World/Planet Earth/' # replace only on line 2 sed '1,3s/World/Planet Earth/' # replace on all lines from 1 to 3 sed '1,$s/World/Planet Earth/' # replace on all lines from first to last ``` --- name: sed3 ## `sed` Address Specifiers - Addresses can also be specified with a .greenbold[regex] in slashes. To illustrate, we introduce another command, the .greenbold[print] command, which is the letter `p`. The print command prints the lines specified by its address specifier. But `sed` by default prints all lines anyway, so we need a way to tell `sed` not to print all lines. The `-n` option tells `sed` .redbold[not to print any lines]. For example this produces no output: ```bash $ echo -e "Hello World\nHello World\nHello World" | sed -n 's/World/Planet Earth/' ``` - The print command will print the lines specified in its address specifier: ```bash $ echo -e "abc \ndef \nghi" | sed -n '1,2p' abc def ``` If the address specifier is a regular expression, it prints those lines that match: ```bash $ echo -e "abc \ndef \nghi" | sed -n '/^def/p' def ``` --- name: sed4 ## `sed` Address Specifiers - The address specifier can be negated as well using `!`, in which case `sed` applies the command to lines that are not matched by the address: ```bash $ echo -e "abc \ndef \nghi" | sed -n '/^def/!p' abc ghi $ echo -e "abc \ndef \nghi" | sed -n '2,$!p' abc ``` --- name: sed-scripts ## Filters: `sed` - `sed` has many commands but we will not cover them here. Others not covered here are .centered_80[ | Command | Description | |:--------- |:---- | | `=` | Output the current line number.| | `a `| Append text after the current line.| | `d `| Delete the current line.| | `i` | Insert text in front of the current line.| | `y/set1/set2/` | Perform transliteration by converting characters from `set1` to the corresponding characters in `set2`. | ] --- name: sed_big_example ## Using `sed` to display the URLS in an HTML file Suppose that I have a file that contains many lines of the form ``` <li><p><a href="some url">general description</a> more text</p></li> ``` as well as many other lines that do not look like that, and I would like to produce a file containing just the urls from those lines. How can I use `sed` to do this easily? I can use a single `sed` command to produce this result. The next few slides show how. --- name: sed-example1 ## Using `sed` to display the URLS in an HTML file 1. We first write a pattern that matches the lines from which we want to extract the URL: ```bash sed -n 's/<li><p>\<a href="\([^"]*\)">.*/\1/p' opensource_links.php ``` The `-n` option tells `sed` not to print any lines from the input: ```bash sed -n ' ' opensource_links.php # prints nothing ``` Without the `-n` option `sed` prints every line, and if you requested a substitution, as in ```bash sed 's/<li><p>/foo/' opensource_links.php ``` it will print every line, substituting as directed each first occurrence of `<li><p>` on a line by `foo`. If you add a `p` flag to the end of the substitution, and put the `-n` option on `sed`, it will only print the lines in which you made a substitution: ```bash sed -n 's/<li><p>/foo/p' opensource_links.php ``` --- name: sed-example2 ## Using `sed` to display the URLS in an HTML file - If we now use as the target pattern `<li><p><a href="\([^"]*\)">.*` this will match all lines that start with the `<li><p><a href="` followed by the URL, followed by `">` and followed by any text at all, and __it will capture the URL in its variable `\1`__. We use that fact to make `\1` the replacement text and add the `p` option: ``` sed -n 's/<li><p>\<a href="\([^"]*\)">.*/\1/p' opensource_links.php ``` which is the needed command. - And we can easily create a Markdown files by this next modification: ``` sed -n 's/<li><p><a href="\([^"]*\)">\([^<]*\)<.*/[\2](\1)\n/p' opensource_links.php ``` --- name: cut_paste1 ## Filters: `cut` and `paste` - Both `cut` and `paste` are handy. `cut` can be used to cut lines in specific places, whether at character positions, or by field positions, and can output the cut pieces with different output delimiters. You can specify what delimits the fields. - The default field separator is the TAB character. - You can use `cut` on csv files to pick out columns: - Examples ```bash cut -f1,5 -d: /etc/passwd ``` prints the first and fifth fields of the `/etc/password` file, i.e., the username and "gcos" field. ```bash cut –c1-10 myfile ``` prints only the first 10 characters of each line of `myfile`. --- name: cut_paste2 ## Filters: `cut` and `paste` (2) - `paste` combines lines consisting of the sequentially corresponding lines from different files, separated by TABs, to standard output. - Example: Suppose that there are two files named `cities` and `countries` whose contents are as shown below (in two columns to save space.) .left-column2[ ```bash $ cat cities Rome Paris London Dublin Tokyo ``` ] .right-column2[ ```bash $ cat countries Italy France England Ireland Japan ``` ] .below-column2[ Then the `paste` command will "merge" the files onto standard output, separating the words with tabs: ```bash $ paste cities countries Rome Italy Paris France London England Dublin Ireland Tokyo Japan ``` ] --- name: cut_paste3 ## Filters: `cut` and `paste` (3) - We can feed standard input into the `paste` command simultaneously. A hyphen '-' represents standard input when it is used in place of a filename argument, as in ```bash paste file - ``` We can use this idea to add numbers to the lines in our previous example, e.g. line 1, 2,3, 4, and 5, as follows: ```bash $seq 1 5 | paste - cities countries 1 Rome Italy 2 Paris France 3 London England 4 Dublin Ireland 5 Tokyo Japan ``` - See what happens when you try ```bash $seq 1 30 | paste - - - ``` Pretty interesting? --- name: awk1 ## `awk` - `awk` is not just an extremely powerful filter; _it is a programming language_. The `awk` filter implements the AWK programming language, named after its authors, __A__ho, __K__ernighan, and __W__einberger. - You give `awk` a program and files on which the program is run, and `awk` runs the program on one file after another. The program is either enclosed in single quotes on the command line, or passed to `awk` in a file like so: ```bash awk [AWK-OPTIONS] -f program-file file ... ``` - If the program is enclosed in single quotes on the command-line, then you run `awk` like this: ```bash awk [AWK-OPTIONS] 'program-text' file ... ``` - Unlike the other filters, `awk` treats each input line as a sequence of fields, delimited by an __input field separator__, by default any amount of whitespace. Fields are named `$1`,`$2`, `$3`, and so on. $0 is the entire line. - If there is no file argument, `awk` reads from standard input. - There is a lot to learn about `awk` - these slides contain just some simple examples. --- name: awk2 ## `awk` (2) - Every `awk` program is a sequence of __pattern-action__ instructions or function definitions*. .footnote[ * There are other kinds of statements as well, not covered here. ] - A pattern-action instruction is of the form ```bash pattern {action} ``` where the pattern can be any of - `BEGIN` - `END` - a regular expression - a comparison - empty and the action is an instruction in a mostly C-like syntax. - Example ```bash awk ' $1 == "reboot" {print $2; }' file1 file2 file3 ``` which prints the second field in any line whose first field is the string "reboot" from the input files `file1`, `file2`, and `file3`. --- name: awk3 ## `awk` (3) - The input field separator can be a character or a regular expression. The command-line option `-F` sets it; use single quotes to protect regular expressions from the shell: ```bash awk -F: '{print $3}' /etc/passwd ``` which separates fields with the colon ":", and ```bash awk -F'aa*' '{print $3}' file1 ``` which separates fields with one or more `a`'s. - Simples uses of `awk` are to print lines with fields that meet a condition, or to print the fields in a different order. `awk` has both a simple `print` instruction as well as all forms of the `C` `printf`: ```bash awk -F, ' {print $1, $2}' names.csv ``` prints the first two fields of every line with a space between them, whereas ```bash awk -F, '{printf "%s\t%s\n", $1, $2}' names.csv ``` prints the first two fields with a TAB between them and a newline after. --- name: awk4 ## `awk` (4) - The __`BEGIN`__ pattern causes `awk` to execute the associated action __before__ reading its input. The __`END`__ pattern's action is executed __after__ all input has been read. The following adds the values of field $1 on all lines and prints their sum. ```bash awk ' BEGIN { sum = 0 } { sum += $1 } END { print sum }' ``` - `awk` has variables that do not need declarations, and - `awk` has operators just like those in C. - The `awk` built-in variable __`NR`__ is the total number of records (lines) read so far. __`NF`__ is the number of fields on the current line, so this command prints the average of the field $1 values: ```bash awk ' BEGIN { sum = 0 } { sum += $1 } END { if ( NR > 0 ) { print sum/NR } }' ``` - This ```bash awk ' { printf "Line %d has %d fields.\n", NR, NF }' ``` displays for each input line, a line of the form ```bash Line 1 has 10 fields. ... ``` --- name: awk5 ## `awk` (5) - The variable `$NF` is the last field on a line, so ```bash awk '{ print $NF }' ``` prints the last field on every input line. - The variable `FNR` is the input record number __in the current input file__. We can print the headings of a CSV file using the following `awk` script: ```bash cat somecsvfile.csv | awk -F, ' \ { if (FNR==1) \ for (i=1; i<=NF; i++) \ printf "Field %d:\t%s\n",i,$i \ }' ``` <!-- / --> The backslashes are needed to prevent the shell from treating each line as a separate command. Notice that `$i` is used to iterate through `$1`, `$2`,... `$NF`. --- name: awk6 ## `awk` (6) - You can use extended regular expression matching in the pattern: ```bash awk -F, ' BEGIN {sum = 0} $1 ~ /[AB].*/ { sum += $3 } END {print sum}' names.csv ``` which adds the values from field $3 for all input lines whose first field starts with an "A" or a "B" from the csv file `names.csv`. All extended regular expressions can be used in `awk`. They must be enclosed in `/ /` brackets. - Logical expressions can be patterns. This `awk` script finds the smallest unused user id in the password file to assign to a new user: ```bash ypcat passwd | \ awk 'BEGIN {FS = ":" ; MAX = 0 } ($3 > MAX ) {MAX = $3} \ END {printf " %s\n", MAX+1} ' ``` The baskslashes are used at the ends of lines that are part of the command. And `FS` is the field separator variable. This is another way to dynamically change it. --- name: awk7 ## `awk` (7) - `awk` has the following control flow statements: - if (condition) statement [ else statement ] - while (condition) statement - do statement while (condition) - for (expr1; expr2; expr3) statement - for (var in array) statement - break - continue - delete array[index] - delete array - exit [ expression ] - { statements } - a switch statement like C's. - `awk` also has many built-in functions, including numeric functions, string functions, time functions, bit manipulationm and more. - It has array variables as well. - There is much more to `awk` than can be described in a few slides. The man page is a good place to look for a comprehensive description of what it can do. --- name: more_filters ## Filters yet to come: - Some filters have not yet made it into these slides. The most important of these are - sed, shuf, split, tr Of these, `sed` is the most powerful, and the hardest to master. The others have a shallow learning curve, and you can read the manpages for them to figure out how to use them. --- name: links ## Useful Links A list of some relevant links - [List of Common Linux Commands](https://ss64.com/bash/) - [Linux Scripting Tutorial](https://bash.cyberciti.biz/guide/Main_Page) - [GNU bash Manual](https://www.gnu.org/software/bash/manual/bash.html) - [Introduction to Text Manipulation in Unix](https://developer.ibm.com/articles/au-unixtext/#25.Resources|outline) --- name: exercise1 ## Exercises 1 Start with some easier ones first. In the exercises, the word _command_ means a structured command - you might need to use pipes or even nested commands. 1. Look at the man page for the `shuf` command. Then write a command that generates a permutation of the integers from 1 to 100 in a file named `permutation100`. 1. Write a command that puts the absolute path names to all C or C++ files in your home directory or below it in a file named `mysourcecode` in your home directory. These files also include header files with a suffix of `.h`. 1. Write a command to display the first 10 lines of every file in the current working directory.For simplicity, assume that the directory contains only plain text files. --- name: exercise2 ## Exercises 2 1. Write a command that prints the number of times that the word "lie" occurs in the set of all files in the current directory with a `.html extension`. Make it case insensitive. 1. This one is not hard after you do a bit of rummaging through the man page for `grep`. Write a command that will print every line of a file preceded by its line number followed by a colon. For example, if the file has two lines ```bash All that glitters is not gold. ``` then it will display ```bash 1: All that glitters 2: is not gold. ``` --- name: exercise3 ## Exercises 3 1. The `ps` command displays process status information for a set of processes running on the local machine. With the `-ef` flags, `ps` lists the status of every process. Look at its output and then write a __script__ named `showprocs` that, when given a user's name, prints the number of processes currently running on that user's behalf. Although the slides so far have not shown the form of a script, you can model it from the following: ```bash #!/bin/bash # Print the first command line argument and exit echo $1 $1 $1 ``` The first line is required in its exact form. The remaining lines that start with `#` are comment lines; you put them there as documentation. The `$1` is a shell variable that stores the first word on the command line after the command itself. `$1` is replaced by the word typed after the script's name when the command is executed. For example if the above script is in a file named `echo3` then we would see the following: ``` $ echo3 hello hello hello hello ``` if we first make the script executable by typing `chmod +x echo3`. --- name: exercise4 ## Exercises 4 1. The `history` command in bash displays the commands you have run recently. The file `~/.bash_history` stores by default the last 500 commands. Inspect that file and then write a command that displays the ten commands you have used the most recently. 1. Before changing a file, it is sometimes safe to make a copy of it and tag the copy with today's date, as for example, by changing `README.md` to `README.md.2019.11.10`. Write a script named `cpdated` that could be used to make a copy with the current date appended to the name, so that ```bash cpdated myfile ``` would create `myfile.2019.11.10` if run on November 10, 2019. Don't worry about error checking such as whether the file exists or whether you have permission to create a file in the current directory. --- name: exercise5 ## Exercises 5 1. Suppose that you want to drop down the headings by one level each in a set of markdown files whose names end in a `.md` extension, in a directory named `documents`. For example, you want a heading starting with `#` to become a `##` heading, and a `##` heading to become a `###` heading. But you do not want level 4 headings to change, so `####` stays as `####`. You are not sure whether there are spaces after the heading tag before the actual text. You can do any of the following: 1. Open a graphical editor and use its __find/replace__ feature on each and every file. 1. Open a command-line editor like `vi` or `vim` and use its find functionality to find every occurrence and change it. 1. Use `vi` or `vim` to do a __global substitution__. 1. Use a well-designed `sed` command to do all of the replacements in a single shot. The last alternative is clearly the best use of your time, and you can ask `sed` to make a backup in case you are nervous about ruining your files. What is the `vi` substitution that will do this? What is the `sed` command that can do this? --- name: exercise6 ## Exercises 6 1. Suppose that for stylistic reasons, you need to replace every C-style single-line comment in your C++ file by C++ style comments. For example, you need to replace ```bash /* The following code finds the min element in the array */ ``` by ```bash // The following code finds the min element in the array ``` regardless of whether there is code to the left of the comment. You can do any of the following: 1. Open a graphical editor and use its __find/replace__ feature. 1. Open a command-line editor like `vi` or `vim` and use its find functionality to find every occurrence and change it. 1. Use `vi` or `vim` to do a __global substitution__. 1. Use a well-designed `sed` command to do all of the replacements in a single shot. The last alternative is clearly the best use of your time, and you can ask `sed` to make a backup in case you are nervous about ruining your files. What is the `vi` substitution that will do this? What is the `sed` command that can do this? --- name: exercise7 ## Exercises 7 1. (HARD) There is a command called `cal` that displays a calendar in Linux: ```bash $ cal April 2019 Su Mo Tu We Th Fr Sa 1 2 3 4 5 6 7 8 9 10 11 12 13 14 15 16 17 18 19 20 21 22 23 24 25 26 27 28 29 30 ``` Using only `echo` and `paste`, try to display the calendar in a similar format for the current month. Hint: if you enclose a sequence of commands in parentheses, they are treated as a single command, e.g.: ```bash $ (echo hello; echo goodbye) | wc 2 2 14 $ echo hello; echo goodbye | wc hello 1 1 8 ```